Problem 4 : Term Weighting Schemes in Information Retrieval

Information retrieval is the process of evaluating a user's query, or information need, against a set of documents (books, journal articles, web pages, etc.) to determine which of the documents satisses the query. With the advent of the World Wide Web, there is suddenly a need to query enormous sets of documents both eeciently and accurately. In the vector space model of information retrieval, documents are represented by sparse vectors each component of which corresponds to a term, usually a word, in the documents set. In the simplest case, the components of these vectors are the raw frequency counts of each term in each document. More sophisticated term weighting schemes are used to improve information retrieval accuracy. We study a speciic term weighting scheme (log-entropy weighting) to determine its eeectiveness on diierent aspects of retrieval. New approaches to term weighting are also examined. In addition, we describe our workshop experience and some of our technical work.